Video display device and video display method

ABSTRACT

A video display device comprising; a display; an exercise amount detection sensor configured to detect an exercise amount of a viewer while the viewer is viewing content displayed and output sensor information; a timer that measures a viewing time; and a processor. The processor: acquires the viewing time from the timer; calculates the exercise amount of the viewer based on the sensor information acquired within a predetermined period of viewing time; compares the exercise amount with an exercise facilitation threshold for determining whether to facilitate exercise for the viewer; displays a specific object within a display area of the display when the exercise amount falls below the exercise facilitation threshold; and moves the specific object from an inside of the display area to an outside thereof in accordance with an external coordinate system expressing a position in a real space associated with a two-dimensional coordinate system in the display area.

TECHNICAL FIELD

The present invention relates to a video display device and a videodisplay method.

BACKGROUND ART

When viewing content on a video display device such as a head mounteddisplay (hereinafter, referred to as “HMD”), a smartphone, a tabletdevice, and the like, a user may suffer from eyestrain caused by keepingthe same posture for a long time. As a technique for eliminating thiseyestrain, Patent Literature 1 discloses “a fatigue recovery aidingapparatus comprising an image generation means for generating an imageof a prescribed object to be moved reciprocatively in the horizontaldirection on a screen of an video display device and a display controlmeans for automatically displaying the image of the object on the screenof the video display device, wherein when a user looking downwardfollows the image of the object with the eyes, feeling of fatigue of theuser can be recovered (excerpted from Abstract)”.

Furthermore, Patent Literature 2 describes “an electronic device whichurges a viewer of content to blink, comprising a blinking informingmeans for informing the viewer of a blinking timing while the viewer isviewing the content, and an input operation detecting means fordetecting an input operation by the viewer while the viewer is viewingthe content, wherein when the input operation detecting means does notdetect the input operation by the viewer continuously for more than afirst predetermined period of time, the blinking informing means informsthe viewer of the blinking timing (excerpted from Abstract)”.

CITATION LIST Patent Literature

-   Patent Literature 1: JP-A-2005-266170-   Patent Literature 2: JP-A-2009-060465

SUMMARY OF INVENTION Technical Problem

In the above-mentioned Patent Literature 1 and Patent Literature 2, whenusing the image display apparatus or the electronic device for a longtime, the viewer is prompted to do exercise of the eye regardless of asituation of the viewer as a way to reduce eyestrain. However, theviewer feels tired not only in the eyes, but also in a wide part of thebody, for example, stiff in the neck and shoulder. Neither PatentLiterature 1 nor Patent Literature 2 takes into consideration thereduction of feeling of fatigue that can occur in a wide part of thebody, and therefore, problems still remain in terms of the above.

The present invention has been made in order to solve the problem asmentioned above, and thus an object of the present invention is toprovide a video display device and a video display method with which aneffect of eliminating feeling of fatigue of a viewer, which includes notonly the fatigue in the eyes but also the one that can occur in a widepart of the body while the viewer is viewing content on the videodisplay device, can be expected.

Solution to Problem

In order to solve the problems as mentioned above, the present inventionincludes the technical features described in the scope of claims. As oneaspect of the present invention, provided is a video display devicecomprising; a display; an exercise amount detection sensor configured todetect an amount of exercise of the video display device while contentdisplayed on the display is viewed, and output sensor information; atimer that measures a viewing time of the content; and a processorconnected to the display, the exercise amount detection sensor, and thetimer, respectively, the processor being configured to: acquire theviewing time from the timer; calculate the amount of exercise of thevideo display device based on the sensor information acquired within apredetermined period of viewing time; compare the amount of exercisewith an exercise facilitation threshold value for determining whether tofacilitate an exercise to a viewer; display a specific object within adisplay area of the display when the amount of exercise is equal to orless than the exercise facilitation threshold value, a display positionof the specific object being defined by a coordinate system in which atwo-dimensional coordinate system expressing a position of a pixel inthe display area of the display is associated with an externalcoordinate system expressing a position in a real space; and move thedisplay position of the specific object from an inside of the displayarea to an outside thereof in accordance with the external coordinatesystem.

Advantageous Effects of Invention

According to the present invention, it is possible to expect an effectof eliminating feeling of fatigue of a viewer, which includes not onlythe fatigue in the eyes but also the one that can occur in a wide partof the body while the viewer is viewing content on the video displaydevice. The objects, configurations, and advantageous effects other thanthose described above will be clarified by explanation of theembodiments below.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 illustrates an appearance of a head mounted display.

FIG. 2 is a hardware configuration diagram illustrating an exemplaryinternal configuration of an HMD.

FIG. 3 is a block diagram illustrating an exemplary functional blockconfiguration of an HMD.

FIG. 4 is a flowchart illustrating an entire processing flow of an HMD.

FIG. 5 is a flowchart illustrating a processing flow of an exercisefacilitation mode.

FIG. 6 illustrates a viewing state in a normal viewing mode.

FIG. 7 illustrates a viewing state in an exercise facilitation mode.

FIG. 8 illustrates an exercise table.

FIG. 9 illustrates an exercise pattern of the determined exercise forrecovery from fatigue.

FIG. 10 illustrates the correspondence between an external coordinatesystem defining a display position of a specific object and atwo-dimensional coordinate system of a display area.

FIG. 11 illustrates an exemplary exercise check list.

FIG. 12 illustrates a state in which a wearer is viewing a video with animmersive HMD.

FIG. 13 illustrates a state in which a wearer is viewing a video with animmersive HMD.

FIG. 14 illustrates a viewing state in the third embodiment.

FIG. 15 is a flowchart illustrating a processing flow of the thirdembodiment.

FIG. 16 is a flowchart illustrating a processing flow of the fifthembodiment.

FIG. 17A illustrates the front surface of a smartphone.

FIG. 17B illustrates the back surface of a smartphone.

FIG. 18 is a flowchart illustrating a processing flow of the sixthembodiment.

FIG. 19 illustrates a case where the face of a wearer is directed to thefront of a smartphone.

FIG. 20 illustrates a case where the face of a wearer is not directed tothe front of a smartphone.

DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments of the present invention will be described withreference to the drawings.

Throughout the drawings, the same features are provided with the samereference signs, and repetitive explanation therefor will be omitted.

First Embodiment

FIG. 1 illustrates an appearance of a head mounted display (HMD) 1according to the present embodiment. The HMD 1 illustrated in FIG. 1 isa see-through type head mounted display. The HMD 1 includes atransmissive or semi-transmissive display 73. A wearer 2 wearing the HMD1 (see FIG. 6 ) is able to see a scene in which a virtual object andvideo displayed on the display 73 is superimposed on the outsidescenery. Although not illustrated, the present embodiment is alsoapplicable to an immersive HMD. The wearer 2 wearing the immersive HMDdoes not directly see the surrounding scene, but sees a virtual world(virtual reality). Furthermore, the immersive HMD employs avideo-through method, that is, uses data of an external field imagecaptured by an out-camera 72 (see FIG. 2 ) to see a video in which avirtual object is superimposed on the external field image. Although thepresent embodiment is applicable to any type described above, in thefollowing, the see-through HMD 1 will be described as an example.

[Example of HMD Hardware Configuration]

FIG. 2 is a hardware configuration diagram illustrating an exemplaryinternal configuration of the HMD 1.

The HMD 1 includes a system bus 3, a main processor 20, a RAM 41, a ROM42, a flash memory 43, a Global Positioning System (GPS) receiver 51, ageomagnetic sensor 52, a range sensor 53, an acceleration sensor 54, agyro sensor 55, a timer 56, a wireless communication I/F 61, a telephonenetwork communication I/F 62, an in-camera 71, the out-camera 72, adisplay 73, a microphone 81, a speaker 82, an audio decoder 83, a buttonswitch 91, and a touch screen 92. These components are connected to thesystem bus 3.

The main processor 20 controls the entire operations of the HMD 1 inaccordance with predetermined operation programs. The main processor 20may be configured with any component such as a CPU, an MPU, or adedicated logic circuitry.

The system bus 3 is a data communication channel that interconnects themain processor 20 with the respective components in the HMD 1. The mainprocessor 20 and the components in the HMD 1 transmit and receivevarious commands, data, and the like through the system bus 3.

The RAM 41 defines a rewritable program work area, which is, forexample, used by the main processor 20 at the time of execution ofvarious programs.

Each of the ROM 42 and the flash memory 43 retains various programs forimplementing the functions of the HMD 1, operation setting values,sensor information including values detected by sensors which will bedescribed later, and various data for display such as virtual objectsand content. The ROM 42 and flash memory 43 are so-called nonvolatilestorages that retain stored data even when power is not supplied to theHMD 1 from the outside.

The flash memory 43 retains operation programs downloaded from a networkand various data created by the operation programs. Through downloadprocessing from each server device on the network, it is possible toupdate and enhance each operation program stored in the flash memory 43.

Furthermore, the flash memory 43 can retain content such as movingimages, still images, and audio data downloaded from a network. Inaddition, the flash memory 43 can retain data such as moving images andstill images captured by the in-camera 71 or the out-camera 72.

The RAM 41, the ROM 42, and the flash memory 43 are examples ofstorages. Other devices, for example, a semiconductor memory elementsuch as a solid state drive (SSD) or a magnetic disk drive such a harddisc drive (HDD) may be used therefor.

The main processor 20 is configured to acquire sensor information fromthe GPS receiver 51, the geomagnetic sensor 52, the range sensor 53, theacceleration sensor 54, and the gyro sensor 55, respectively, and alsoacquire a viewing time measured by the timer 56. Then, the mainprocessor 20 uses the sensor information and the viewing time to detectthe position, tilt, direction, movement, and the like of the HMD 1.Furthermore, the HMD 1 may further include other sensors, such as anilluminance sensor, a proximity sensor, an altitude sensor, and thelike.

The wireless communication I/F 61 is connected to a network such as theInternet through a wireless access point and the like, and transmits andreceives data to and from each server on the network. The wirelesscommunication I/F 61 may be connected to the wireless access point andthe like via, for example, Wi-Fi (registered trademark) or BlueTooth(registered trademark).

The telephone network communication I/F 62 is connected to acommunication network through a base station using a mobilecommunication network to transmit and receive data to and from a serveron a communication network according to the third-generation mobilecommunication system (hereinafter, referred to as “3G”) such as GlobalSystem for Mobile Communications (GSM, registered trademark), WidebandCode Division Multiple Access (W-CDMA), CDMA 2000, or Universal MobileTelecommunications System (UMTS). Similarly, the telephone networkcommunication I/F 62 is connected to the communication network accordingto the Long Term Evolution system (LTE), or a communication method suchas so-called the fourth generation system (4G) or fifth generationsystem (5G).

Each of the wireless communication I/F 61 and telephone networkcommunication I/F 62 includes an encoding circuitry, a decodingcircuitry, an antenna, and the like.

Furthermore, the HMD 1 may include other communication interfaces suchas an infrared communication I/F.

Each of the in-camera 71 and the out-camera 72 converts the lightreceived through a lens into an electric signal using an electronicdevice such as a charge coupled device (CCD) or a complementary metaloxide semiconductor (CMOS) sensor to input image data of thesurroundings and that of an object.

The display 73 is, for example, a display device such as a liquidcrystal panel, and provides the wearer 2 of the HMD 1 with image data.The HMD 1 includes a video RAM (not illustrated). A virtual object orvideo is displayed on the screen of the display 73 based on the imagedata received by the video RAM.

The microphone 81 converts the voice of the wearer 2 into audio data andinputs the audio data.

The speaker 82 outputs sound and the like.

The audio decoder 83 performs decoding processing of an encoded audiosignal as necessary.

Each of the button switch 91 and touch screen 92 is an operation deviceused to input an operation instruction to the HMD 1. The operationdevice is not limited to the button switch 91 and the touch screen 92.For example, an operation signal to the HMD 1 may be transmitted from aseparate portable device (for example, a smartphone or a tablet device)connected thereto by wired communication or wireless communication. Uponreceiving the operation signal, the HMD 1 is operated in accordance withthe received operation signal. Alternatively, based on the audio dataconverted by the microphone 81, the main processor 20 may execute soundrecognition processing to generate an operation signal to control theoperation of the HMD 1.

A part of the exemplary configuration of the HMD 1 illustrated in FIG. 2is not essential to the present embodiment. Accordingly, even if theconfiguration of the HMD 1 does not include such an unessentialcomponent, the advantageous effects of the present embodiment are notimpaired. In this connection, the configuration of the HMD 1 may furtherinclude components which are not illustrated, for example, a digitalbroadcast reception function and an electronic money settlementfunction.

[Functional Block of HMD 1]

FIG. 3 is a block diagram illustrating an exemplary functional blockconfiguration of the HMD 1. The main processor 20 loads a program storedin the ROM 42 or the flash memory 43 onto the RAM 41 and executes theprogram, whereby each functional block unit illustrated in FIG. 3 isimplemented.

A main controller 30 controls the operations of the HMD 1.

A sensor information acquisition unit 31 is configured to acquire sensorinformation from each of the GPS receiver 51, the geomagnetic sensor 52,the range sensor 53, the acceleration sensor 54, and the gyro sensor 55.The sensor information acquisition unit 31 stores the acquired sensorinformation in a sensor information storage unit 32 configured with awork area of the storage (for example, the flash memory 43). The sensorinformation acquisition unit 31 is also configured to acquire a viewingtime measured by the timer 56, and thus an exercise amount analysis unit33 and an exercise amount monitoring unit 35 can refer to the viewingtime.

The exercise amount analysis unit 33 is configured to read the sensorinformation for each predetermined period of time stored in the sensorinformation storage unit 32, and analyze the movement amount, movementspeed, and movement direction of the wearer 2 to generate the amount ofexercise information. The exercise amount analysis unit 33 stores theamount of exercise information in an exercise amount information storageunit 34 configured with a work area of the storage (for example, theflash memory 43).

The exercise amount monitoring unit 35 is configured to read the amountof exercise information stored in the exercise amount informationstorage unit 34, and accumulate the amount of exercise information for aset time. Then, the exercise amount monitoring unit 35 monitors anexercise condition of the wearer 2 of the HMD 1 based on the cumulativevalues.

The exercise facilitation unit 36 is configured to control display ofimages for facilitating the wearer 2 of the HMD 1 to do the exercisebased on the values accumulated by the exercise amount monitoring unit35.

[Processing Mode of HMD 1]

FIG. 4 is a flowchart illustrating an entire processing flow of the HMD1.

The processing of the HMD 1 is roughly divided into a normal viewingmode, an exercise amount monitoring mode, and an exercise facilitationmode following the exercise amount monitoring mode. The normal viewingmode is a mode in which the wearer 2 of the HMD 1 can freely operate theHMD 1 to enjoy the content and applications. The exercise amountmonitoring mode is a mode for monitoring the amount of exercise of thehead of the wearer 2 of the HMD 1 during the normal viewing mode so asto cause mode shift to the exercise facilitation mode. The exercisefacilitation mode is a mode for facilitating an exercise for recoveryfrom fatigue to eliminate feeling of fatigue as needed. When the mainpower is turned on, the HMD 1 executes the normal viewing mode and theexercise amount monitoring mode in parallel.

[Normal Viewing Mode]

As a normal viewing mode, for example, content display processing willbe described as an example. The main processor 20 of the HMD 1 stands byuntil it receives a content display start instruction from the buttonswitch 91 or the touch screen 92 (step S101: NO). Upon receiving inputof the content display start instruction (step S101: YES), the mainprocessor 20 starts the content display processing (step S102).

[Exercise Amount Monitoring Mode]

The main controller 30 resets the timer 56 for measuring a viewing timeT of the content to 0, and also resets values of cumulative amount ofexercise M1, M2, M3 to 0 (step S201). In the present embodiment, as theamount of exercise of the head of the wearer 2 of the HMD 1 (see FIG. 6), the amount of rotational, parallel, and vertical movement of the headis measured. Accordingly, as the sensor information for measuring theamount of exercise of the head of the wearer of the HMD 1, inparticular, the geomagnetic sensor 52, the acceleration sensor 54, andthe gyro sensor 55 are used. A first cumulative amount of exercise M1represents a cumulative value of the sensor information from thegeomagnetic sensor 52, a second cumulative amount of exercise M2represents a cumulative value of the sensor information from theacceleration sensor 54, and a third cumulative amount of exercise M3represents a cumulative value of the sensor information from the gyrosensor 55.

The sensor information acquisition unit 31 acquires the sensorinformation from the GPS receiver 51, the geomagnetic sensor 52, therange sensor 53, the acceleration sensor 54, and the gyro sensor 55, andstores the acquired information in the sensor information storage unit32. In addition, the timer 56 starts to measure the viewing time T (stepS202).

The exercise amount analysis unit 33 reads, from the sensor informationstorage unit 32, the sensor information from the geomagnetic sensor 52,that from the acceleration sensor 54, and that from the gyro sensor 55,calculates M1, M2, M3, respectively, and stores the calculation resultsas the amount of exercise information in the exercise amount informationstorage unit 34.

The exercise amount monitoring unit 35 reads, from the storage, a firstlower limit of amount of exercise (M1th), a second lower limit of amountof exercise (M2th), and a third lower limit of amount of exercise(M3th), which are preset for M1, M2, M3, respectively. The exerciseamount monitoring unit 35 also reads M1, M2, M3 from the exercise amountinformation storage unit 34. Then, the exercise amount monitoring unit35 executes determination processing with the following equations (1) to(3) (step S203).

M1>M1th  (1)

M2>M2th  (2)

M3>M3th  (3)

In the case where at least one of the determination results of theequations (1) to (3) is positive (step S203: YES), the processingreturns to step S201. In this case, it is estimated that the head of thewearer of the HMD 1 is not so motionless as to require exercisefacilitation.

On the other hand, in the case where all the determination results ofthe equations (1) to (3) are negative (step S203: NO), the processingmoves to step S204. In this case, it is estimated that the amount ofexercise of the head of the wearer 2 of the HMD 1 is so small that theexercise facilitation is necessary.

The exercise amount monitoring unit 35 further executes thedetermination processing with the following equation (4) using theviewing time T and a preset allowable time threshold value Tth (stepS204).

T>Tth  (4)

In the case where the determination result of the equation (4) isnegative (step S204: NO), the processing returns to step S202. In thiscase, it is estimated that the amount of exercise of the head of thewearer 2 of the HMD 1 is so small that the exercise facilitation isnecessary, however, since the small amount of exercise state does notexceed the allowable time threshold value Tth, it is determined that theexercise facilitation is not currently required.

On the other hand, in the case where the determination result of theequation (4) is positive (step S204: YES), the normal viewing mode isinterrupted (step S103), and the mode is shifted to the exercisefacilitation mode (step S300). In this case, the state in which theamount of exercise of the head of the wearer 2 of the HMD 1 is smallcontinues for a long time, and accordingly, it is estimated that doingthe exercise should be facilitated to the wearer 2 of the HMD 1.

[Exercise Facilitation Mode]

FIG. 5 is a flowchart illustrating a processing flow of the exercisefacilitation mode.

The exercise facilitation unit 36 is configured to display an object(hereinafter, referred to as a “specific object”) to be used in theexercise for recovery from fatigue on a display area 100 of the display73 (step S301).

FIG. 6 illustrates a viewing state in the normal viewing mode.

FIG. 6(a) illustrates a scene in which the wearer 2 of the HMD 1 isviewing in the normal viewing mode. In the normal viewing mode, content111 is displayed on the display area 100 of the display 73. The wearer 2views the content 111 which is superimposed on the real scenery.

FIG. 6(b) illustrates the relation between a position of the wearer 2 inthe three-dimensional space of the real space and a position at whichthe content 111 is virtually viewed. The content 111 to be displayed inthe normal viewing mode is the content that the wearer 2 wishes to vieweven while he or she is in motion, and thus it is displayed at a fixedposition with respect to the display area 100.

Furthermore, if the content 111 is virtually arranged in thethree-dimensional space of the real space, the virtual object isarranged and fixed with respect to the three-dimensional coordinates inthe real space. For example, the wearer 2 can view a virtual object 113of a cup fixed to a table 112 only when the wearer 2 moves the head andthe virtual object 113 is on the extension of the display area 100 andthe line of sight of the wearer 2.

Thus, in FIG. 6 , the wearer 2 cannot view the virtual object 113.

FIG. 7 illustrates a viewing state in the exercise facilitation mode.

The exercise facilitation unit 36 is configured to display the specificobject 120 within the display area 100 of the display 73 (FIG. 7(a)).FIG. 7(b) illustrates the relation between a position of the wearer 2 inthe three-dimensional space of the real space and a position at whichthe specific object 120 is virtually viewed. Note that the position andangle of the head of the wearer 2 in FIG. 7(b) shall not change from theposition and angle of the head of the wearer 2 in FIG. 6(b).

In the following, FIG. 5 will be referred to again. The exercisefacilitation unit 36 refers to an exercise list illustrated in FIG. 8 ,and determines an exercise pattern to be use in the exercise forrecovery from fatigue, which is to be facilitated to the wearer 2 (stepS302).

FIG. 8 illustrates an exercise table. In the exercise table, as thetypes of exercise of the head, a rotating exercise, a laterally turningexercise, and a vertically turning exercise are registered. Furthermore,in the exercise table, a direction toward which the wearer 2 is made toturn initially, a direction of rotating, the speed, and the frequency ofexercise are set. Here, an example in which the speed is set to anintermediate speed level in the case where the speed can be set in fivelevels is described.

There are several algorithms used by the exercise facilitation unit 36to determine the exercise for recovery from fatigue. For example, thealgorithm to be used may be determined, as a result of the equations (1)to (3), so as to interpolate the amount of exercise that is equal to orless than an exercise facilitation threshold value. More specifically,the exercise facilitation unit 36 analyzes the values of M1, M2, M3, andcalculates the direction and amount of exercise of the head of thewearer 2. Then, the type, order, and frequency of exercise to be used inthe exercise for recovery from fatigue may be determined in order fromthe exercise direction with the small amount of exercise.

FIG. 9 illustrates an exercise pattern of the determined exercise forrecovery from fatigue. In the exercise for recovery from fatigue, thetype of exercise selected from the exercises registered in FIG. 8 ,order, and frequency of times to be done are defined.

The exercise facilitation unit 36 moves a display position of thespecific object 120 within the display area 100 in accordance with theexercise pattern defined in the exercise for recovery (step S303).

FIG. 10 illustrates the correspondence between an external coordinatesystem defining a display position of the specific object 120 and atwo-dimensional coordinate system of the display area 100.

The exercise facilitation unit 36 acquires the current position of theHMD 1 from the GPS receiver 51. This current position is defined in aGPS coordinate system, that is, a three-dimensional coordinate system ofthe real world, and is referred to as an external coordinate system inthe present description. In this connection, in the case where theaccurate current position cannot be obtained by the GPS receiver 51, forexample, while the user is indoors, the current position may be obtainedby adding a travel distance calculated based on the information such asfrom the acceleration sensor 54 mounted on the HMD 1 to the point wherethe accurate current position acquired by the GPS receiver 51 wasrecorded. Furthermore, the position of the HMD 1 at the time of beingactivated may be processed as the origin of the external coordinatesystem, and then be corrected after the accurate current position isacquired by the GPS receiver 51. In the following, FIG. 10 illustrates adisplay example of the one eye for the purpose of explanation. On theother hand, in the case of three-dimensional display by both the eyes,the specific object 120 in the three-dimensional coordinate system ofthe real world is converted into a display area of each eye and thendisplayed.

The exercise facilitation unit 36 generates calibration data forconverting the two-dimensional coordinate system (st-coordinate system)of the display area 100 into the external coordinate system(xyz-coordinate system) by making pixels P0, P1, P2, P3 within thedisplay area 100 associated with a coordinate system in which thetwo-dimensional coordinate system within the display area 100 isassociated with the external coordinate system expressing a position inthe real space.

-   -   P0: (s₀, t₀) (x₀, y₀, z₀)    -   P1: (s₁, t₀) (xi, y₀, z₀)    -   P2: (s₀, t₂) (x₀, y₂, z₀)    -   P3: (s₁, t₂) (xi, y₂, z₀)

The exercise facilitation unit 36 moves the display position of thespecific object 120 along the external coordinate system. At this time,the exercise facilitation unit 36 converts the coordinates of thedisplay position of the specific object 120 in the external coordinatesystem into the two-dimensional coordinates of the display area 100using the calibration, whereby the specific object can be displayed at adesired position in the external coordinate system.

Alternatively, the exercise facilitation unit 36 may be configured todisplay the specific object 120 on the display 73 in the case where thedisplay position of the specific object 120 expressed by the externalcoordinate system is on the normal vector V of a surface including thedisplay area 100, which passes through pixels of the display area 100.In this case, when the display position of the specific object 120 isnot on the normal vector V of any point in the two-dimensionalcoordinate system, the specific object 120 is not displayed on thedisplay.

The exercise facilitation unit 36 applies the coordinates of P0, P1, P2,P3 in the external coordinate system to the following equation (5) toobtain the equation defined in the external coordinate system of thesurface including the display area 100.

ax+by+cz=d  (5)

Thereafter, the exercise facilitation unit 36 obtains, based on theequation (5), normal vectors V0, V1, V2, V3 of the surface including thedisplay area 100, which pass through the points P0, P1, P2, P3,respectively. The case where the specific object 120 is in athree-dimensional space 101 surrounded by V0, V1, V2, V3 means that thespecific object 120 is present on the line of sight of the wearer 2which passes through the display area 100, and accordingly, the exercisefacilitation unit 36 causes the specific object 120 to be displayed inthe display area 100 in order to allow the wearer 2 to see it.

As illustrated in FIG. 10 , when the specific object 120 is moved fromthe inside of the display area 100 toward the outside in the leftdirection of FIG. 10 and thus deviates from the three-dimensional space101, the wearer 2 cannot see the specific object 120. Thus, when theexercise facilitation unit 36 moves the display position of the specificobject 120 along the external coordinate system, with the result thatthe specific object 120 starts to deviate from the inside of thethree-dimensional space 101, as indicated by the specific object120_out, the specific object 120 becomes gradually invisible within thedisplay area 100 starting from a portion that has deviated from thethree-dimensional space 101. That is, in the case where pixels thatcorrespond to the coordinates of the display position of the specificobject 120 in the external coordinate system are present within thedisplay area 100, the exercise facilitation unit 36 causes the specificobject 120 to be displayed on the pixels, and in the case where nopixels that correspond to the coordinates of the display position of thespecific object 120 in the external coordinate system are present withinthe display area 100, the exercise facilitation unit 36 does not causethe specific object 120 to be displayed on the display 75.

FIG. 7(c) illustrates a state in which the specific object 120 is movedfurther upward than that in FIG. 7(a). In FIG. 7(d), the wearer 2 movesthe head in an elevation direction (namely, doing the exercise for head)and accordingly, the display area 100 of the HMD 1 also moves relativeto the real space. Thus, the wearer 2 is looking up at the specificobject 120.

The exercise facilitation unit 36 determines whether the inclinationangle (for example, elevation angle) θ of the head of the wearer 2 isequal to or more than a predetermined achievement determinationthreshold value θth based on the following equation (6) (step S304).

θ≥θth  (6)

FIG. 11 illustrates an example of an exercise check list. In theexercise check list, the achievement determination threshold value θthused to determine whether the exercise has been achieved in view of theangle of the head is defined for each point of each exercise.

If the equation (6) is negative (step S304: NO), the processing returnsto step S303, and the exercise facilitation unit 36 continues moving thespecific object. If the equation (6) is positive (step S304: YES), itcan be estimated that the exercise has been done, and thus the exercisefacilitation unit 36 ends the subsequent exercise or the exercise forrecovery from fatigue per se. If all of the exercise patterns andfrequency of times defined in FIG. 9 are not completed (step S305: NO),the exercise facilitation unit 36 selects the subsequent movementdirection (namely, changes the movement direction) (step S306), andmoves the specific object 120 toward the movement direction that hasbeen selected (step S303). In the above, a display example for makingthe wearer 2 face upward has been described, and similarly, the exercisefacilitation unit 36 can prompt the wearer 2 to move the head by movingthe specific object 120 downward, leftward, or rightward.

On the other hand, if all the exercise patterns and frequency of timesof the exercise for recovery from fatigue which has been determined inFIG. 9 are completed (step S305: YES), the exercise facilitation unit 36finishes the exercise for recovery from fatigue. The main processor 20restarts the normal viewing mode (step S101), and also restarts theexercise amount monitoring mode (step S201).

According to the present embodiment, while the wearer 2 is viewing thecontent 111 in the normal viewing mode, the amount of exercise of thehead of the wearer 2 is monitored, and if the state in which the amountof exercise is small exceeds the allowable time threshold value Tth, themode is shifted to the exercise facilitation mode. Then, moving thedisplay position of the specific object 120 forcibly, that is,regardless of the operation instruction from the wearer 2 causes thewearer 2 to track the specific object 120 with the eyes, therebygenerating the movement of the head of the wearer 2. Furthermore, bychecking the amount of exercise of the head of the wearer 2 about,whether the exercise for recovery from fatigue is properly achieved canbe confirmed. As a result, when the wearer 2 concentrates on viewing thecontent 111 of the HMD 1 and thus the amount of exercise of the headdecreases, it is possible to facilitate the exercise of the head andneck more effectively.

Second Embodiment [Immersive HMD Image]

Each FIG. 12 and FIG. 13 illustrates a state in which the wearer 2 isviewing a video with the immersive HMD 1. In the case of the immersiveHMD 1, the wearer 2 sees a scenery in the real world in which anexternal image captured by the out-camera 72 is displayed within thedisplay area 100 of the display 73. Furthermore, the wearer 2 can see avirtual object and content superimposed on the external image. FIG.12(a) illustrates a situation in which the wearer 2 is viewing theexternal image showing a partial area of the real world captured by theout-camera 72, which is displayed within the display area 100. FIG.12(b) illustrates an example of the posture of the head of the wearer 2who is seeing the video of FIG. 12(a).

The difference between the first embodiment and the second embodiment isthat, while the first embodiment uses a virtual object as the specificobject 120, the second embodiment does not use a virtual object as aspecific object but uses an object in which an external image is reducedand displayed as a specific object 121 (see FIG. 13(a)).

When the mode is shifted to the exercise facilitation mode, the exercisefacilitation unit 36 generates the specific object 121 by graduallyreducing, from the size corresponding to the size of the display area100, an image obtained immediately before the shifting to the exercisefacilitation mode. Then, in the same manner as the first embodiment, theexercise facilitation unit 36 moves the specific object 121. Thespecific object 121 may be a reduced image of the content displayed inthe normal viewing mode, or may be a reduced image of an external imagewhen the content is not displayed in the normal viewing mode and onlythe external image is being displayed.

The exercise facilitation unit 36 darkens a peripheral portion(background area) of the specific object 121 in order to provide aluminance difference between the specific object 121 and the peripheralportion. In the case where the image of the specific object 121 is dark,the luminance difference therebetween may be provided by graduallyincreasing the brightness of the entire of the specific object 121 whilereducing the size thereof, thereby allowing the wearer 2 to recognizethe specific object 121 more easily.

Using the image obtained by reducing the immediately preceding externalimage as the specific object 121 enables the wearer 2 to do the exercisefor recovery from fatigue without interrupting viewing of the content orthe external image.

Third Embodiment

In the exercise for recovery from fatigue, the exercise facilitationunit 36 moves the specific objects 120, 121 toward the outside of thedisplay area 100. This may cause the wearer 2 to fail to follow themovement of the specific objects 120, 121 and thus lose track thereof.In the third embodiment, guidance is provided in the case where thewearer 2 loses track of the specific objects 120, 121. In the following,the case using the specific object 120 will be described as an example.

FIG. 14 illustrates a viewing state in the third embodiment. In FIG. 14, the specific object 120 is not inside the display area 100, whichcorresponds to a state in which the specific object 120 is locatedoutside the three-dimensional space 101 surrounded by V0, V1, V2, V3 inFIG. 10 . This state makes the wearer 2 unsure which direction, namely,left, right, front, or rear, to face in order to view the specificobject 120.

In this regard, the exercise facilitation unit 36 is configured to, whendetermining that the specific object 120 deviates from the display area100, perform processing to notify the wearer 2 of the direction in whichthe specific object 120 is present.

For example, the exercise facilitation unit 36 displays a guide 130 fornotifying the direction of the specific object 120. FIG. 15 is aflowchart illustrating a processing flow of the third embodiment, whichis the processing of notifying a direction of a specific object. Thesame processes as those in FIG. 5 are provided with the same referencesigns, and explanation therefor will be omitted. In step S310, theexercise facilitation unit 36 determines whether the specific object iswithin the display area 100. If the specific object 120 is within thedisplay area 100, the processing proceeds to step S304, however, if thespecific object 120 is not within the display area 100, the exercisefacilitation unit 36 calculates a relative position between the specificobject 120 and the display area 100 in step S311. Next, in step S312,the exercise facilitation unit 36 displays, as illustrated in FIG. 14 ,the guide 130 which is movement assist information showing the movementfor the direction in which the head should be moved.

Furthermore, the exercise facilitation unit 36 may be configured torearrange the specific object 120 to be within the display area 100, andmove the rearranged specific object 120 again in a direction in whichthe wearer 2 should face. In this case, the exercise facilitation unit36 determines that the wearer 2 could not follow the speed of movementof the specific object 120, and accordingly, may set the speed less thanthe speed of the initial movement so that the wearer 2 can follow thespecific object 120 more easily.

Still further, the exercise facilitation unit 36 may be configured tonotify the direction using the sound from the speaker 82 provided on theHMD 1. In the case where the speaker 82 is a three-dimensional speakerthat allows the wearer 2 to recognize the direction of the sound source,the exercise facilitation unit 36 may notify the wearer 2 of theposition of the specific object 120 by emitting the sound from thedirection of the specific object 120.

According to the present embodiment, even when the wearer 2 loses trackof the specific object 120, the wearer 2 is notified with the movementdirection of the specific object 120 by the guide 130. This enables thewearer 2 to find the specific object 120, and thus do the exercise forrecovery from fatigue.

Fourth Embodiment

The fourth embodiment is an embodiment for facilitating recovery fromfatigue of the eyeballs of the wearer 2. In the fourth embodiment, thein-camera 71 captures images of the eye of the wearer 2. Then, theexercise amount monitoring unit 35 detects the movement of the point ofview of the wearer 2 based on the images captured by the in-camera 71.Within a period of time in which to decide whether the exercise forrecovery from fatigue is necessary, the exercise amount monitoring unit35 determines whether a cumulative value Meye of the movement of thepoint of view of the wearer 2 is equal to or less than an exercisefacilitation threshold value Meyeth which is predetermined for themovement of the point of view.

When Meye is equal to or less than Meyeth, the exercise facilitationunit 36 generates the specific object 120 and moves the display positionthereof. A moving range of the display position may be kept within thedisplay area 100 since it is sufficient that a point-of-view exercisecan be performed therein.

Since the point of view of the wearer 2 moves less from the displayposition of the content while the wearer 2 is viewing the content, theeyes of the wearer 2 may be fatigued. According to the presentembodiment, the in-camera 71 is used to track the movement of the pointof view of the wearer 2 so as to facilitate the exercise for recoveryfrom fatigue of the eyeballs as necessary, whereby the reduction of theeyestrain of the wearer 2 can be expected.

Furthermore, the exercise facilitation unit 36 may move the specificobject 120 in a direction away from the body of the wearer 2 to make thewearer 2 walk or extend his or her hands as the exercise for recoveryfrom fatigue. For example, the exercise facilitation unit 36 may use thevirtual object 113 of the cup illustrated in FIG. 6(a) as a specificobject, and provide the wearer 2 with a sound guidance to cause thewearer 2 to hold the virtual object 113. Thus, the wearer 2 is made toexercise by moving the head to see the virtual object 113 that was notvisualized at the time of hearing the sound guide and extending the handtoward the virtual object 113. In addition, the exercise facilitationunit 36 may provide the wearer 2 with a sound guide to cause the wearer2 to catch the specific object 120 shaped like a fairy, which makes thewearer 2 stand up and walk.

Fifth Embodiment [Determination of Shift to Exercise Facilitation Mode]

Depending on the situation of the wearer 2, there are cases thatshifting to the exercise facilitation mode is not appropriate. In thefifth embodiment, whether shifting to the exercise facilitation mode isappropriate is determined.

FIG. 16 is a flowchart illustrating a processing flow of the fifthembodiment.

When the determination in step S204 is positive, the exercise amountmonitoring unit 35 acquires at least one of the sensor information, theexternal image, or the audio data from at least one of the varioussensors, the out-camera 72, or the microphone 81 to determine thesituation of the wearer 2.

For example, when determining that the wearer 2 is in motion based onthe sensor information from the GPS receiver 51 and the accelerationsensor 54, the exercise amount monitoring unit 35 does not shift themode to the exercise facilitation mode since the exercise for recoveryfrom fatigue may put the wearer 2 in danger, and thus continues thenormal viewing mode and the situation determination processing (stepS205: NO).

Furthermore, for example, when determining that the wearer 2 is inconversion with someone based on the external image from the out-camera72 or the audio data from the microphone 81, the exercise amountmonitoring unit 35 does not shift the mode to the exercise facilitationmode since it is an inappropriate timing to facilitate the exercise forrecovery from fatigue, and thus continues the normal viewing mode andthe situation determination processing (step S205: NO).

When the situation which has been determined as being inappropriate iseliminated (step S205: YES), the exercise amount monitoring unit 35shifts the mode to the exercise facilitation mode.

According to the present embodiment, the exercise amount monitoring unit35 determines the situation of the wearer 2, and starts the exercisefacilitation mode by the exercise facilitation unit 36 only when thesituation is appropriate. Thus, the exercise for recovery from fatiguecan be facilitated under the situation where there is no problem if thewearer 2 does the exercise for recovery from fatigue.

In the above, the embodiment additionally including the determinationprocessing of whether shifting to the exercise facilitation mode ispossible has been described. On the other hand, for example, the wearer2 may use the button switch 91 or the touch screen 92 to avoid theshifting to the exercise facilitation mode.

Sixth Embodiment

The sixth embodiment is an embodiment in which the present invention isapplied to a mobile video display device other than the HMD 1, forexample, a smartphone 210 or a tablet device.

FIG. 17A illustrates the front surface of the smartphone 210, and FIG.17B illustrates the back surface of the smartphone 210.

On the front surface of the smartphone 210, the display 73 equipped withthe touch screen 92, the in-camera 71 capable of taking a selfie, andthe speaker 82 are provided. On the back surface of the smartphone 210,the out-camera 72, a photographing light 74 such as an LED, and themicrophone 81 are provided. Inside the housing of the smartphone 210, inthe same manner as the HMD 1, various sensors are provided to detect theorientation of the body of the smartphone 210. In addition, although notillustrated, a range sensor used in the three-dimensional authenticationof the face of the wearer 2 may be arranged on the front surface of thesmartphone 210, and furthermore, a range sensor used to detect thedistance to the subject or the shape of the subject may be arranged onthe back surface.

FIG. 18 is a flowchart illustrating a processing flow according to thesixth embodiment. The processing according to the sixth embodimentdiffers from the processing according to the first embodiment in that,after moving the display position of the specific object 120 in stepS303, the exercise facilitation unit 36 analyzes the movement directionof the smartphone 210, and compares at least one of an inclination θs ofthe smartphone 210 and the amount of movement Ms thereof withpredetermined threshold values θsth and Msth (step S401). If theinclination or amount of movement of the smartphone 210 is less than themovement determination threshold values (step S401: NO), the position ofthe smartphone 210 relative to the position of the wearer 2 correspondsto the movement only within the movement determination threshold value.This causes the exercise facilitation unit 36 to determine that thewearer 2 is following the specific object 120 with the eyes, andaccordingly, the processing returns to step S303 and the exercisefacilitation unit 36 continues moving the display position of thespecific object 120.

When determining that the inclination or amount of movement of thesmartphone 210 is equal to or more than the movement determinationthreshold values (step S401: YES), the exercise facilitation unit 36determines whether the wearer 2 is correctly facing the smartphone 210(step S402). In the case of the smartphone 210, unlike the case of theHMD 1, the movement of the smartphone 210 does not match the movement ofthe head of the wearer 2, and accordingly, it is necessary to determinewhether the head of the wearer 2 is in motion.

The exercise facilitation unit 36 causes the in-camera 71 to capture animage of the face of the wearer 2 to determine the orientation of theface. The exercise facilitation unit 36 causes the in-camera 71 tocapture the image of the front of the face of the wearer 2, for example,before start of the exercise processing or during viewing of thecontent, and compares the image of the front of the face of the wearer 2with the image of the face of the wearer 2 captured during the exerciseprocessing so as to determine whether the front of the face of thewearer 2 is directed to the front of the smartphone 210.

When determining that the orientation of the face of the wearer 2 isdirected to the front of the smartphone 210 (step S402: YES), theexercise facilitation unit 36 determines whether the exercise pattern iscompleted (step S305), and if it is completed, shifts the mode to thenormal viewing mode and the exercise amount monitoring mode.

On the other hand, when determining that the orientation of the face ofthe wearer 2 is not directed to the front of the smartphone 210 (stepS402: NO), the exercise facilitation unit 36 performs warning processing(step S403). In the warning processing, for example, attention to promptthe wearer 2 to move the head is displayed in the display area 100, orsound is used to prompt the wearer 2 to move the head.

FIG. 19 illustrates the case where the front of the face of the wearer 2is directed to the front of the smartphone 210. FIG. 19(a) illustratesthe image displayed within the display area 100 of the smartphone 210,and at this time, the front of the face of the wearer 2 is directed tothe display area 100 of the smartphone 210 as illustrated in FIG. 19(b).In FIG. 19(a), the specific object 120 is within the screen, and thesmartphone 210 is also directed toward a predetermined direction.

FIG. 20 illustrates the case where the face of the wearer 2 is notdirected to the front of the smartphone 210. In FIG. 20(b), the face ofthe wearer 2 is not directed to the front of the smartphone 210, andthus it is not determined that the wearer 2 is properly doing theexercise. In this case, as illustrated in FIG. 20(a), a warning display450 is displayed.

Furthermore, steps S401 to S403 are repeated until the smartphone 210and the orientation of the face of the wearer 2 reach predeterminedpositions, and then the processing proceeds to step S305 of determiningwhether the exercise pattern is completed.

According to the present embodiment, even in the case of a portablevideo information device such as the smartphone 210 whose movement doesnot match the movement of the head, it is possible to facilitate theexercise for recovery from fatigue every certain period of time and toconfirm whether the exercise for recovery from fatigue is correctlydone.

In the embodiment above, instead of the in-camera 71, a range sensorarranged on the front surface of the smartphone 210 may be used todetect the orientation of the face based on the position of the eyes,nose, or mouth of the wearer 2, and determine the positional relationbetween the face of the wearer 2 and the smartphone 210 based on thedetected information.

Each of the embodiments above are an example of an embodiment of thepresent invention. The present invention is not limited to theembodiments above, and various modifications are possible. For example,a part of an exemplary embodiment may be replaced with a part of otherembodiments, and a part of other embodiments may be added to theexemplary embodiment. These modifications belong to the scope of thepresent invention. Furthermore, the numerical values, messages, and thelike appearing in the description and drawings are merely examples, andthe effect of the present invention is not impaired even if differentones are used.

A part of or all of the functions and the like of the present inventiondescribed above may be implemented by hardware, for example, bydesigning it by an integrated circuitry. Alternatively, a microprocessorunit or the like may interpret and execute a program for implementingeach function or the like, thereby implementing it by software. Hardwareand software may be used together. The software may be stored in advancein a program unit of an HMD, a mobile video display device, or the likeat the time of product shipment. The software may be acquired fromvarious servers or the like on the Internet after the product isshipped. Furthermore, the software which is provided in a memory card,an optical disk, or the like may be acquired.

Furthermore, the control lines and information lines which areconsidered to be necessary for the purpose of explanation are indicatedherein, but not all the control lines and information lines of actualproducts are necessarily indicated. It may be considered that almost allthe configurations are actually connected to each other.

REFERENCE SIGNS LIST

-   -   1: HMD    -   2: wearer    -   2 a: viewer    -   3: system bus    -   20: main processor    -   30: main controller    -   31: sensor information acquisition unit    -   32: sensor information storage unit    -   33: exercise amount analysis unit    -   34: exercise amount information storage unit    -   35: exercise amount monitoring unit    -   36: exercise facilitation unit    -   41: RAM    -   42: ROM    -   43: flash memory    -   51: GPS receiver    -   52: geomagnetic sensor (exercise amount detection sensor)    -   53: range sensor    -   54: acceleration sensor (exercise amount detection sensor)    -   55: gyro sensor (exercise amount detection sensor)    -   56: timer    -   61: wireless communication I/F    -   62: telephone network communication I/F    -   71: in-camera    -   72: out-camera    -   73: display    -   74: photographing light    -   81: microphone    -   82: speaker    -   83: audio decoder    -   91: button switch    -   92: touch screen    -   100: display area    -   101: three-dimensional space    -   111: content    -   112: table    -   113: virtual object    -   120: specific object    -   120_out: specific object    -   121: specific object    -   130: guide    -   210: smartphone    -   450: warning display

1. A video display device comprising; a display; an exercise amountdetection sensor configured to detect an amount of exercise of the videodisplay device, and output sensor information; a timer that measurestime; and a processor connected to the display, the exercise amountdetection sensor, and the timer, respectively, the processor beingconfigured to: acquire the time from the timer; calculate the amount ofexercise of the video display device based on the sensor informationacquired within a predetermined period of time; compare the amount ofexercise with an exercise facilitation threshold value for determiningwhether to facilitate an exercise to a viewer; display a specific objectwithin a display area of the display when the amount of exercise isequal to or less than the exercise facilitation threshold value, adisplay position of the specific object being defined by a coordinatesystem in which a two dimensional coordinate system expressing aposition of a pixel in the display area of the display is associatedwith an external coordinate system expressing a position in a realspace; and move the display position of the specific object from aninside of the display area to an outside thereof in accordance with theexternal coordinate system, determine an exercise achievement of thevideo display device based on the sensor information.
 2. The videodisplay device according to claim 1, wherein the processor is configuredto notify a direction in which the video display device is to be movedso that the specific object is visualized when the specific object isoutside the display area.
 3. The video display device according to claim1, wherein the processor is configured to: set the speed of the specificobject less in a case where the specific object deviates from thedisplay area.
 4. The video display device according to claim 1, whereinthe processor is configured to: acquire the sensor information outputfrom the exercise amount detection sensor after displaying the specificobject, and calculate the amount of exercise after displaying thespecific object; and compare the amount of exercise after displaying thespecific object with an achievement threshold value for determiningwhether the viewer has done the exercise,
 5. The video display deviceaccording to claim 1, wherein the processor is configured to: determinewhether the viewer is in motion based on the sensor information; and notto display the specific object when determining that the viewer is inmotion.
 6. The video display device according to claim 1, furthercomprising at least one of a microphone and an out-camera for capturingan image of an external scenery of the video display device, wherein theprocessor is configured to: determine whether the viewer is inconversation based on at least one of audio information collected by themicrophone and an external image captured by the out-camera; and not todisplay the specific object when determining that the viewer is inconversation.
 7. The video display device according to claim 1, whereinthe video display device is a head mounted display, and the exerciseamount detection sensor is at least one of a geomagnetic sensor, a rangesensor, an acceleration sensor, or a gyro sensor.
 8. The video displaydevice according to claim 1, wherein the video display device is amobile video display device, the mobile video display device furthercomprises an in-camera for capturing an image of the viewer who isviewing the display, the processor is further connected to thein-camera, and the processer is configured to: compare the amount ofexercise after displaying the specific object with an achievementthreshold value for determining whether the viewer has done theexercise, and in a case where determining the amount of exercise afterdisplaying the specific object is equal to or more than the achievementthreshold value, and also determining that the viewer is directly facingthe display, as a result of determining whether the viewer is directlyfacing the display based on the image captured by the in-camera, stop todisplay the specific object.
 9. The video display device according toclaim 1, wherein the processor is configured to display the specificobject formed with a virtual object.
 10. The video display deviceaccording to claim 1, wherein the processor is configured to display thespecific object formed with a reduced image in which a display size ofthe content is reduced.